System and method for diagnostic vector classification support

ABSTRACT

The diagnostic vector classification support system and method disclosed herein may both reduce the time and effort required to train radiologists to interpret medical images, and provide a decision support system for trained radiologists who, regardless of training, have the potential to miss relevant findings. In an embodiment, a morphological image is used to identify a zone of interest in a co-registered functional image. An operator&#39;s grading of a feature at least partially contained within the zone of interest is compared to one or more computer-generated grades for the feature. Where the operator and computer-generated grades differ, diagnostic support can be provided such as displaying additional images, revising the zone of interest, annotating one or more displayed images, displaying a computer-generated feature grade, among other possibilities disclosed herein.

This application is a divisional of U.S. patent application Ser. No.14/205,005, filed Mar. 11, 2014, which claims benefit of U.S.Provisional Patent Application No. 61/799,213 filed Mar. 15, 2013,entitled “Vector Classification in an Optoacoustic Imaging System;” U.S.Provisional Patent Application No. 61/810,238 filed Apr. 9, 2013,entitled “Functional Modality with Histopathologic Correlation in anOptoacoustic Imaging System;” and U.S. Provisional Patent ApplicationSer. No. 61/898,392 filed Oct. 31, 2013, entitled “Functional UltrasoundModality in Optoacoustic Breast Imaging.” The entire disclosures ofthose applications are incorporated herein by this reference.

This application includes material which is subject to copyrightprotection. The copyright owner has no objection to the facsimilereproduction by anyone of the patent disclosure, as it appears in thePatent and Trademark Office files or records, but otherwise reserves allcopyright rights whatsoever.

FIELD

The present invention relates in general to the field of medicalimaging, and in particular to system relating to support forinterpretation of optoacoustic imaging.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, and advantages of theinvention will be apparent from the following more particulardescription of preferred embodiments as illustrated in the accompanyingdrawings, in which reference characters refer to the same partsthroughout the various views. The drawings are not necessarily to scale,emphasis instead being placed upon illustrating principles of theinvention.

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the U.S. Patent and TrademarkOffice upon request and payment of the necessary fee.

FIG. 1 is a schematic block diagram illustrating an embodiment of asystem for use in support of diagnostic vector classification oflesions.

FIG. 2 is a flow diagram illustrating an embodiment of a process fordiagnostic vector classification support.

FIG. 3 shows an optoacoustic image with boundary curves displayedthereon in accordance with an embodiment of the invention.

FIG. 4 is a six image display illustrating an embodiment of image datafor use in support of diagnostic vector classification of lesions.

FIG. 5 is a six image display with boundary curves presented thereon inaccordance with an embodiment of the invention.

FIG. 6 is a diagram illustrating an embodiment of a graphical userinterface for use in operator feature grading and lesion classification.

FIG. 7 shows six optoacoustic, combined map images illustrating examplesof a feature internal vascularity in accordance with an embodiment ofthe subject invention.

FIG. 8 shows six optoacoustic, combined map images illustrating examplesof a feature internal blush in accordance with an embodiment of thesubject invention.

FIG. 9 shows six optoacoustic, hemoglobin map images illustratingexamples of a feature internal hemoglobin in accordance with anembodiment of the subject invention.

FIG. 10 shows seven optoacoustic images illustrating examples of afeature presence of capsular or boundary zone vessels in accordance withan embodiment of the subject invention.

FIG. 11 shows six optoacoustic images illustrating examples of a featurepresence of peripheral vessels in accordance with an embodiment of thesubject invention.

FIG. 12 shows six optoacoustic, combined map images illustratingexamples of a feature interfering artifacts in accordance with anembodiment of the subject invention.

FIGS. 13A and 13B show the categories and rankings of various featuresin accordance with an embodiment of the subject invention.

FIGS. 14A and 14B show a scatter plot of various features in accordancewith an embodiment of the subject invention.

FIG. 15 shows feature vectors with the strongest correlation betweenfeatures in accordance with an embodiment of the subject invention.

While the invention is amenable to various modifications and alternativeforms, specifics thereof have been shown by way of example in thedrawings and will be described in detail. It should be understood,however, that the intention is not to limit the invention to theparticular embodiments described. On the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention.

DETAILED DESCRIPTION

The following description and drawings are illustrative and are not tobe construed as limiting. Numerous specific details are described toprovide a thorough understanding. Yet, in certain instances, well-knownor conventional details are not described in order to avoid obscuringthe description. References to one or an embodiment in the presentdisclosure are not necessarily references to the same embodiment; and,such references mean at least one.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed which may be exhibited by some embodiments and not by others.Similarly, various requirements are described which may be requirementsfor some embodiments but not other embodiments.

As used in this description and in the following claims, “a” or “an”means “at least one” or “one or more” unless otherwise indicated. Inaddition, the singular forms “a,” “an,” and “the” include pluralreferents unless the content clearly dictates otherwise. Thus, forexample, reference to a composition containing “a compound” includes amixture of two or more compounds.

As used in this specification and the appended claims, the term “or” isgenerally employed in its sense including “and/or” (that is, both theconjunctive and the subjunctive) unless the context clearly dictatesotherwise.

The recitation herein of numerical ranges by endpoints includes allnumbers subsumed within that range (e.g. 1 to 5 includes 1, 1.5, 2,2.75, 3, 3.80, 4, and 5).

Unless otherwise indicated, all numbers expressing quantities ofingredients, measurement of properties and so forth used in thespecification and claims are to be understood as being modified in allinstances by the term “about,” unless the context clearly dictatesotherwise. Accordingly, unless indicated to the contrary, the numericalparameters set forth in the foregoing specification and attached claimsare approximations that can vary depending upon the desired propertiessought to be obtained by those skilled in the art utilizing theteachings of the present invention. At the very least, and not as anattempt to limit the scope of the claims, each numerical parametershould at least be construed in light of the number of reportedsignificant digits and by applying ordinary rounding techniques. Anynumerical value, however, inherently contains certain errors necessarilyresulting from the standard deviations found in their respective testingmeasurements.

The systems and methods are described below with reference to, amongother things, block diagrams, operational illustrations and algorithmsof methods and devices to process optoacoustic imaging data. It isunderstood that each block of the block diagrams, operationalillustrations and algorithms and combinations of blocks in the blockdiagrams, operational illustrations and algorithms, can be implementedby means of analog or digital hardware and computer programinstructions.

Computer program instructions described herein can be provided to aprocessor of a general purpose computer, special purpose computer, ASIC,or other programmable data processing apparatus, such that theinstructions, which execute via the processor of the computer or otherprogrammable data processing apparatus, implements the functions/actsspecified in the block diagrams, operational block or blocks and oralgorithms.

Furthermore, the embodiments of methods presented and described asflowcharts in this disclosure are provided by way of example in order toprovide a more complete understanding of the technology. The disclosedmethods are not limited to the operations and logical flow presentedherein. Alternative embodiments are contemplated in which the order ofthe various operations is altered and in which sub-operations describedas being part of a larger operation are performed independently.

In some alternate implementations, the functions/acts noted in theblocks can occur out of the order noted in the operationalillustrations. For example, two blocks shown in succession can in factbe executed substantially concurrently or the blocks can sometimes beexecuted in the reverse or a differing order, depending upon thefunctionality/acts involved.

Diagnostic Vector Classification Support System

Radiology is a medical specialty that employs the use of imaging todiagnose and/or treat disease visualized within the human body. Aradiologist interprets images created by any of a variety of medicalimaging technologies, and produces a report of findings, impressionand/or diagnosis. Radiologists are highly trained at interpreting one ormore of the types of images created by various medical imagingtechnologies.

Optoacoustic imaging is a relatively new clinical field. Substantialtime and effort is required to train a radiologist to interpret imagescreated from optoacoustic data. The diagnostic vector classificationsupport system and method disclosed herein may both reduce the time andeffort required to train a radiologist to interpret images created fromoptoacoustic data, and provide a decision support system for trainedradiologists who, regardless of training, have the potential to missrelevant findings. While the system described herein is shown withrespect to images created from optoacoustic data, and specificallyimages created from ultrasound and optoacoustic data, it is not solimited, and is equally applicable to other types of medical images.

Turning first to FIG. 1, an embodiment of a diagnostic vectorclassification support system 100 is generally shown. In an embodiment,the system 100 is embodied as a processing subsystem of an imagingsystem, such as the multimodal optoacoustic and ultrasound systemdescribed in U.S. patent application Ser. No. 13/507,222, filed Jun. 13,2013 and entitled “System and Method for Producing Parametric Maps ofOptoacoustic Data” (hereinafter the “Parametric Map Application”). In anembodiment, the system 100 is implemented on a standalone system orgeneral purpose computer, comprising the appropriate software and a userinterface as described herein, adapted to process images produced by oneor more separate imaging systems including separate or multimodaloptoacoustic and or ultrasound systems. In this latter case, the imagesmust be acquired from a suitable source of the images, or transferred tothe system, e.g., via the Internet or by a storage medium and reader.

In an embodiment, a co-registration sub-system 103 obtains a pluralityof images of a volume of tissue and spatially aligns the images. Suchimages may include images produced by various imagining technologiesincluding but not limited to MRI, CT Scan, X-ray, Ultrasound,Optoacoustic, among other modalities. In an embodiment, as shown,structural images, such as those produced by ultrasound are spatiallyaligned with functional images, such as those produced by optoacousticimaging. In embodiment, multiple optoacoustic images or parametric mapsare spatially aligned. In an embodiment, the co-registration sub-system103 is not required because the images obtained by the system 100 arealready spatially aligned. In an embodiment, only portions of the imagesare spatially aligned. In an embodiment, the images are spatiallyaligned with known landmarks or annotations. For more detaileddescription of co-registration techniques, reference can be had to theParametric Map Application.

In an embodiment, the spatially aligned images are received by adiagnosis support sub-system 105. In the embodiment as shown, thediagnosis support sub-system 105 is capable of presenting images andother output to an observer via a display device 107. In an embodiment,the display device 107 comprises a video monitor, screen, holographicdisplay, printer, or other technology known in the art capable ofpresenting two and three dimensional images. In an embodiment, sound,haptic, or other output methods known in the art are used to conveyinformation. In an embodiment, videos may be presented comprising bothsound and images. In an embodiment, the diagnosis support sub-system 105is capable of presenting information via multiple display devices.

In the embodiment as shown, the diagnosis support sub-system 105 is alsocapable to receiving classifications, scoring or other input from theobserver or other operator via an input device 109. In an embodiment,the input device 105 comprises a pointing device such as a mouse,trackball, touch screen, or other pointing device. In an embodiment, theinput device 105 comprises a keyboard, key pad, or other device fortextual input. In an embodiment, the input device 105 comprises amicrophone or other audio input device. Other input devices may be used.In an embodiment, the diagnosis support sub-system 105 is capable ofreceiving input from multiple input devices.

In an embodiment, the diagnosis support sub-system 105 identifies atleast one feature of at least one image that is significant fordiagnosis of a disease or condition. In an embodiment, the operatorprovides input to identify the feature. For example, the operator mayselect one or more pixels in at least one image corresponding to astructure or other significant region of the volume. As used throughoutthis specification and the below claims, the term “corresponding to”means an element of an image or parametric map spatially represents orprovides information about a location or region in a volume of tissue,which term encompasses estimating and approximating of the location orregion. Feature identification is further discussed below through theexample of identification of a lesion or tumor in a volume of tissue.

In an embodiment, the diagnosis support sub-system 105 provides aqualitative or quantitative analysis of the at least one feature. In anembodiment, the operator or other user also provides a qualitativeanalysis of the at least one feature. In an embodiment, the results ofthe system's analysis are compared with the operator's conclusions. Theoperator may also provide additional input either before or aftersystem's evaluation. For example, in an embodiment, the operatorchanges, augments or corrects the system's feature identification. In anembodiment, the operator confirms the system's evaluation. In anembodiment, the system displays additional images or other informationto the operator before the additional input is received. For example,the system may display additional images of the volume (including imagesfrom other modalities), annotations highlighting or otherwise indicatingimage features, additional analysis of the feature, examples (e.g.,images) of the feature as presented in other volumes of tissue, orevaluations of the feature obtained by different algorithms, models, orsystems.

In an embodiment, the at least one feature comprises a lesion. In anembodiment, the at least one feature includes a feature vector includinga plurality of features of the lesion. In embodiment, features found inan interior zone of an image corresponding to an interior region of thelesion are evaluated. In an embodiment, features found in an exterior orexternal zone of an image corresponding to a region of the volumeexterior to the lesion are evaluated.

In an embodiment, images are segmented into three or more regions. Inthe example shown in FIG. 3, an optoacoustic image of a volume of tissueis divided into three regions using two boundary curves. The white,“tumoral” boundary curve defines an interior zone of the imagecorresponding to an interior region of the tumor. In an embodiment, theinterior region of the tumor is defined by the central nidus of thetumor. In an embodiment, the central nidus is hypoechoic on ultrasoundand the interior zone is identified on one or more ultrasound images ofthe volume, which can then be co-registered with optoacoustic images orother parametric maps of the volume.

FIG. 3 also includes a blue, “peritumoral” boundary curve correspondingto a periphery of a peritumoral region of the tumor adjacent to thecentral nidus. In an embodiment, a portion of an image between a tumoraland a peritumoral boundary curve is referred to as a “boundary zone” ofthe image. In an embodiment, more than one boundary zone may exist andcorrespond to regions of the volume outside but adjacent to the centralnidus of a tumor. Boundary zones may not exist adjacent to each edge ofthe tumoral boundary curve. In an embodiment, the peritumoral boundarycurve overlaps with the tumoral boundary curve where no separateboundary zone exists.

In an embodiment, the peritumoral and tumoral boundary curves are usedto define at least three zones of an image corresponding to at leastthree regions of the volume: (1) an interior zone corresponding to theinterior of the tumor; (2) a boundary zone corresponding to aperitumoral region of the volume adjacent to the tumor; and (3) anperipheral zone corresponding to a region of the volume outside both thetumoral and peritumoral regions of the tumor. A feature vector mayinclude features from one or more of these zones. Such features mayinclude, by way of example and not limitation: the internal vascularityof the lesion; internal deoxygenation of the lesion; the peritumoralboundary vascularity of the lesion; the peritumoral deoxygenation of thelesion; the internal de-oxygenated blush; the internal total blood; theexternal peritumoral radiating vessels; and the presence of one or moreinterfering artifacts among other possible features of the lesion. Asfurther discussed below, in an embodiment, these and other features mayappear and be evaluated on ultrasound images, optoacoustic images,parametric maps or other spatial representations of a volume of tissue.

In an embodiment, the diagnosis support sub-system 105 evaluates atleast one feature of the lesion by analyzing one or more images or mapsof the volume. In an embodiment, the diagnosis support sub-system 105develops a qualitative or quantitative value based on its evaluation ofthe feature. In an embodiment, the evaluated features are part of afeature vector associated with the volume. In an embodiment, theevaluation of a feature comprises scoring the feature by trained orotherwise known feature grades.

In an embodiment, the diagnosis support sub-system 105 is adapted topermit identification of a lesion within the volume, to obtain operatorand computer classification of the lesion, to compare theclassifications, and to provide diagnostic support where the computerand operator classifications differ. In an embodiment, diagnosticsupport is provided even where the computer and operator reach the sameclassifications.

In an embodiment, the computer classification of the lesion is based onevaluation of a vector of features of the volume associated with thelesion. In an embodiment, the diagnosis support sub-system 105 guidesthe user through a process for evaluating multiple features of thevolume within the feature vector. In an embodiment, the diagnosissupport sub-system 105 presents information about each of the featuresin parallel (e.g., using different portions of the display device 107).An example user interface is provided in FIG. 6. In an embodiment, thediagnosis support sub-system 105 presents information about each of thefeatures in series. For example, in embodiment, the diagnosis supportsub-system 105 causes the display device 107 to highlight or otherwiseannotate portions of images of the volume that the diagnosis supportsub-system 105 used to evaluate each feature in the volume. Thediagnosis support sub-system 105 may then solicit the user's evaluationof the feature or other input.

The user may use the sub-system 105's annotating to reach the user's ownconclusions about each feature. In an embodiment, the sub-system 105displays such annotations in response to the user's evaluation when theuser's evaluation differs or differs substantially from the sub-system105's evaluation of a feature. In an embodiment, the user input maycomprise a correction of the zone of an image that the diagnosis supportsub-system 105 used to evaluate one or more features in the featurevector. In an embodiment, for example, the user may correct or augmentone of the boundary curves used to define the periphery of the lesionand the diagnosis support sub-system 105 re-evaluates on or morefeatures of the lesion based on the corrected periphery.

In an embodiment, the diagnosis support sub-system 105 displays orotherwise presents its own evaluation of one or more features to a uservia the display device 107. In an embodiment, the diagnosis supportsub-system 105 displays or highlights the features of the images or mapsanalyzed to produce an evaluation. In an embodiment, the diagnosissupport sub-system 105 presents this information to the user in responseto input from the user. In an embodiment, the user inputs the user's ownevaluations of the feature, via the input device 109, before or afterthe diagnosis support sub-system 105 presents its evaluations. In anembodiment, the sub-system 105 displays its evaluations when the user'sevaluation differs or differs substantially from the sub-system 105'sevaluation.

In an embodiment, the diagnosis support sub-system 105 may also displayother information that may be helpful to the user in evaluating thefeature. For example, the sub-system 105 may display images a subjectmatter expert or other actor previously used to evaluate the samefeature (in this or other volumes of tissue). In an embodiment, thesub-system 105 may display images that produced an evaluation similar tothe sub-system 105's evaluation of the feature. In the embodiment, thesub-system 105 may display images that produced an evaluation similar tothe user's evaluation of the feature. In an embodiment, the sub-system105 then solicits input from the user regarding the feature. Forexample, in an embodiment, the sub-system 105 asks the user to confirmor change the sub-system 105's evaluation of the feature.

In an embodiment, the diagnosis support sub-system 105 computes afeature vector for a lesion based on the sub-system 105's and/or theuser's evaluations of features in the vector. In embodiment, the resultsof the sub-system 105's computation are presented to the user. In anembodiment, the results include a suggested classification of thelesion. In an embodiment, the user inputs the user's own classificationof the lesion before or after the sub-system 105 presents itsclassification. In an embodiment, the user is given the opportunity toconfirm or modify the sub-system 105's classification of the lesion. Ifthe user's classification of the lesion differs or differs substantiallyfrom the sub-system 105's classification, the sub-system 105 may presentadditional information and/or solicit additional user input. In anembodiment, the sub-system 105 only presents such additional informationor solicits such additional user input of the difference between theoperator's and the subsystem's feature grades would change thesub-system's classification of the feature.

Obtaining Images

Turning to FIG. 2, an embodiment of a diagnostic vector classificationsupport method 200 is generally shown. At 203, an image is obtained froma source of such images, such as a multimodal optoacoustic andultrasound system such as one described in the Parametric MapApplication. In an embodiment, the image data may comprise one image. Inan embodiment, the image data may comprise a plurality of images. Mostof the examples shown herein are two-dimensional images or maps;however, the systems and methods discussed herein may also be applied tothree or more dimensional representations of a volume of tissue.

In an embodiment where the image data is made up of a plurality ofimages, it is convenient to have the plurality of images co-registered.In an embodiment, the image data comprises radiological information of avolume of tissue. In an embodiment, the images of the image data depictvisible functional or morphological structures in the volume (as theyare available to be depicted by the modality of each image). In anembodiment, the image data comprises six images as generally reflectedin FIG. 4.

FIG. 4 illustrates six co-registered two-dimensional images: onecomprising image information derived from ultrasound 410 (“ultrasound”);one comprising image information derived from optoacoustic imaging, andrepresentative of the response of a longer predominant wavelength oflight 420 (“long wavelength image”); one comprising image informationderived from optoacoustic imaging, and representative of the response ofa shorter predominant wavelength of light 430 (“short wavelengthimage”); and three being multimodal images comprising image informationderived from optoacoustic imaging, one being parametrically reflectiveof total hemoglobin 440 (“total hemoglobin map”), one beingparametrically reflective of deoxygenated hemoglobin 450 (“relativeoptoacoustic map”); and one being parametrically reflective ofdeoxygenated hemoglobin 450 masked using the image 440 parametricallyreflective of total hemoglobin 460 (“combined optoacoustic map”). Formore detailed description of the six image types, references can be hadto the Parametric Map Application.

Identification of a Lesion/Image Segmentation

In an embodiment, at 205, a lesion or tumor is identified in the imagedata obtained at 203. The process of identifying a lesion in an imagemay vary depending on the type of image obtained for classification. Inan embodiment, generally speaking, the goal is to define a perimeter,and potentially a periphery of a lesion as accurately as possible.Proper identification of a lesion perimeter aids in determination ofwhether a finding is internal or external to the lesion. In anembodiment, morphological images, such as ultrasound, are used toidentify features corresponding to structures in a volume of tissue.Such features can then be used to segment the morphological image. In anembodiment, such segmentation is then applied to co-registered spatialrepresentations of the same volume.

In an embodiment, the image(s) are segmented into two or more regions orzones. In an embodiment, segmentation involves outlining or otherwiseidentifying the boundaries of the lesion. In an embodiment, a lesion inan ultrasound image (e.g., conventional ultrasound image 410) may besegmented. In an embodiment, a lesion in an optoacoustic image may besegmented (e.g., images 420, 430, 440, 450, 460). Generally, an imageneeds to contain sufficient information to be capable of beingsegmented.

In an embodiment, segmentation is done by a trained operator, such as,for example, a radiologist. In an embodiment, the image or parts of theimage are displayed on a computer screen and a trained operatorcarefully segments the boundary of the lesion on the display (e.g., bydrawing or manipulating a mouse or other pointing device to select atleast one point on or near the boundary), relying on the data present inthe image. In an embodiment, multiple, co-registered images may be usedas the source of information for segmentation and the trained operatorcan rely upon the data from, e.g., multiple images and/or modalities, todetermine segmentation. In an embodiment, a first boundary isidentified. In an embodiment, the first boundary is a tumoral boundaryas described with reference to FIG. 3.

FIG. 5 illustrates the same images shown in FIG. 4, however the imagesin FIG. 5 each include a white curve representing a tumoral boundary.The boundaries can also be represented by points, dashed lines, or otherannotations. Note that while an operator may have identified the tumoralboundary, e.g., on the ultrasound image 410, a diagnostic vectorclassification support system can display the tumoral boundary on otherimages (e.g, images 420, 430, 440, 450, 460) as shown here. In anembodiment, an operator may provide a rough approximation of the lesion(e.g., a square, rectangle, circle, triangle, incomplete set of points,or freehand sketch) rather than carefully identifying its boundaries.Where an the operator provides such a rough segmentation, as describedin more detail below, the boundaries of the lesion can be more preciselyestimated through the use of a segmentation technique implemented in thediagnostic vector classification support system. In an embodiment, ifthe diagnostic vector classification support system refines, adjusts orotherwise changes the approximation of the user, the user can furtherrefine, adjust or otherwise change the diagnostic vector classificationsupport system's results, leaving the ultimate selection of the boundaryto the operator.

In an embodiment, a second boundary is identified outside the firstboundary. In the embodiment shown in FIG. 5, a blue curve also appearsin each image approximating a peritumoral boundary in the depictedimage. The techniques discussed above with respect to identification ofthe first boundary may also be applied to identification of the secondboundary. As above, although the second boundary curve may have beenidentified only on one of the images (e.g., ultrasound), the diagnosticvector classification support system can display the second boundary onmultiple images. In an embodiment, the second boundary curve is used todefine a peripheral region of the lesion between itself and the firstboundary curve. As with the first boundary, in an embodiment, the secondboundary can be drawn or otherwise identified by an operator, or may begenerally (e.g., roughly) identified and made more precise by the use ofa computer implemented segmentation technique. In an embodiment, thesecond boundary is identified automatically by a computerized process.In an embodiment, the second boundary is defined in relation to thefirst boundary. In an embodiment, the second boundary is a fixeddistance away in the outward normal direction of the first boundary. Theoutward normal direction of the first boundary at a given point on theboundary is perpendicular to the tangent vector of the first boundary atthe given point, such that in most circumstances the outward normaldirection will point away from the interior region of the regionenclosed by the first boundary.

In an embodiment, as with the first boundary, if the diagnostic vectorclassification support system itself identifies the second boundary, orif it refines, adjusts or otherwise changes the user's identification ofthe second boundary, the user can further refine, adjust or otherwisechange the diagnostic vector classification support system's results,leaving the ultimate identification of the second boundary to theoperator.

In an embodiment, the region inside the tumoral boundary is referred toas the interior region (i.e. internal zone), and the region outside thetumoral boundary but inside the peritumoral boundary is the peritumoralor boundary zone. The region outside the peritumoral boundary isreferred to as the peripheral region.

Returning to FIG. 3, the image has been segmented into three zones. Thewhite “tumoral” curve segments the image into an internal zonecorresponding to the interior or central nidus of the represented tumorand an external zone corresponding to regions of the represented volumeexternal to the central nidus of the represented tumor. The blue“peritumoral” curve further segments the external (or exterior) zoneinto a boundary (or peritumoral) zone and a peripheral zone. In anembodiment, the boundary zone corresponds to a portion of therepresented volume adjacent to but outside the central nidus of thetumor. In an embodiment, the boundary zone varies in thickness and canbe absent along some surfaces of the tumor. In an embodiment, theperitumoral boundary curve corresponds to a thick hyperechoic halo thatcan be identified on ultrasound images of the volume. In an embodiment,the peripheral zone corresponds to portions of the volume external toboth the central nidus of the tumor and the boundary zone. In anembodiment, the peripheral zone is further from the central nidus thanthe boundary zone in an outward normal direction with respect to thetumor. In the image shown in FIG. 3, a think yellow line annotates afeature within the peripheral zone.

In an embodiment, features are evaluated which fall within various zonesof the obtained images corresponding to various regions of therepresented volume. In an embodiment, a feature is considered to fallwithin a particular zone if the feature is partially contained withinthat zone. So, for example, in an embodiment, a boundary zone featuremay extend from the boundary zone into the peripheral zone. Or astructure considered to be in the peripheral region of a volume oftissue may extend into a peritumoral region of the volume.

In an embodiment, the boundary zone is considered an important source ofinformation pertaining to classification of a tumor for at least threereasons: (1) it is where the tumor grows and invades surrounding tissue;(2) it is where the host response tries to stop the tumor fromspreading; and (3) it is where cancer cells can convert some host cells(fibroblasts and macrophages) into cancer cells thereby helping thecancer grow. Further, the boundary zone may feature radiating feedingarteries and draining veins that supply the tumor with blood and oxygenand remove wastes from the tumor. Sometimes these vessels areparasitized native vessels and sometimes they are tumor neovessels.

In an embodiment, the boundary zone is very complex and may have manycontributors to its appearance including: proliferating and invadingtumor cells; a rich network of tumor neovessels, most of which areoriented near a 90 degree angle relative to the surface of the tumor(these neovessels are sometimes referred to a boundary zone “whiskers”);tumor associated collage type 3 fibers, which are also orientedperpendicular to the surface of the tumor; tumor associated macrophages;native lymphocytes sent to control the tumor; desmoplasia—fibrous tissuebuilt by the host to create a wall around the tumor; edema—caused byfluid from abnormal tumor vessels; or proteinaceous debris from abnormaltumor vessels. A thin boundary or capsule zone may correspond to abenign lesion, while a thick boundary zone indicated by a thickechogenic halo may correspond to an invasive lesion.

In most cases of invasive cancer, a trained technician can identify theboundary zone on ultrasound images because it differs in echogenicityfrom both the central hypoechoic nidus and from the surrounding normaltissue. Echogenicity can be thought of as a mechanical property of thetissue. Features of co-registered optoacoustic images may also helpidentify the boundary zone in some cases. For example, some optoacousticimages show differences in the boundary zone (or capsule zone) ofmalignant lesions when compared to benign regions: Malignant lesionstend to have short perpendicular oriented tumor neovessels termed“boundary zone whiskers,” while benign lesions tend to exhibit eitherthe complete absence of boundary zone or capsular vessels or have longcurved vessels oriented parallel to the surface of the tumor or capsule,rather than the more perpendicular orientation of most malignantvessels. Capsular vessels tend to be close in or touching the outer edgeof the central nidus. Boundary zone vessels tend to be shorter and moreperpendicular in orientation. In some cases, capusular vessels may bewithin about 1 to 3 mm of the central nidus. In other cases, capsularvessels may appear further away from the central nidus. Peripheralvessels may also appear farther out than the boundary zone. Peripheralvessels generally do not touch the central nidus and may or may nottouch the boundary zone. Peripheral vessels generally radiate from thecentral nidus in a direction roughly perpendicular to the surface of thecentral nidus. Examples of possible boundary zone whiskers are annotatedwith orange lines in FIG. 5. Examples of possible radiating vessels areannotated with yellow lines in FIG. 5.

In an embodiment, as shown in FIG. 5, two closed boundary curves arecalculated or otherwise obtained that completely define a zone of animage. In this case, the boundary zone can be defined by subtracting thezone defined by the inner boundary from the zone defined by the outerboundary. In an embodiment, first and second boundaries obtained mayonly define a portion of the lesion's interior and periphery. Forexample, in an embodiment, first and second boundaries only define theupper portion of a lesion—that is, the portion of the lesion closer tothe sensor. Such an embodiment may be necessitated where the entirelesion does not appear in each of the images or where insufficientdetail to identify the boundaries is found due to a decrease ofinformation available below a certain depth of the lesion.

In an embodiment, one open and one closed boundary curve may beobtained. In an embodiment, the open boundary curve can be closed byconnecting its end-points. In an embodiment, where the closed curverepresents the tumoral boundary, as shown in FIG. 3, various methods canbe used to connect the open boundary curve to the tumoral boundary. Forexample, in an embodiment, image context is used to draw connectingcurves from the open boundary curve to the tumoral boundary. In anembodiment, connecting lines are drawn from each end-point of theboundary curve to the closest point on the tumoral boundary curve. In anembodiment, connecting lines are drawn from each end-point of theboundary curve that intersect the tumoral boundary curve at aperpendicular angle. Where multiple points on the tumoral boundary curvemeet these criteria, the connecting point may be selected in variousways including the point closest or furthest from the end-point, thepoint closest or furthest from the center of the tumor, the point thatcreates the connecting line most perpendicular to the surface of thetumor.

In another embodiment, two open boundary curves may be obtained. In anembodiment, where two open boundary curves are obtained, the tumoralboundary curve can be closed by connecting its end-points or anothertechnique known in the art, and then one or more of the techniquesdiscussed above can be applied. In an embodiment, for each end-point ofthe peritumoral boundary curve, a line is drawn to the closest end-pointof the tumoral boundary curve. In an embodiment, image context can beused to select the connecting points. Other techniques known in the artthat can be applied. In an embodiment, first connecting lines are drawnusing one or more of the techniques discussed above; and image contextis then used to correct the straight connecting lines, which maytherefore become connecting curves.

Displaying Image Data

In an embodiment, at 207, image data is displayed to an operator foranalysis and input. In an embodiment, as discussed above, image data maybe displayed for the purpose of segmentation. In an embodiment, imagedata is displayed or re-displayed after segmentation along with curves,dotted lines, highlights, or other annotations indicating one or moreboundaries used to segment the images. In an embodiment, an operator mayadjust one or more boundary curves at this time via the input device109.

In an embodiment, image data is displayed by means of a display devicesuch as display device 107. In an embodiment, image data may comprisemultiple co-registered images of the volume as discussed above. In anembodiment, examples of feature representations from the same or othervolume of tissue are displayed for comparison. In an embodiment,features from prior imaging of the same patient are displayed forprogress or trend analysis. For example, prior imaging of the samepatient can be displayed to track the progression of a disease such ascancer including tumor classification, growth, vascularity, total blood,and other features. In an embodiment, canonical examples of featuresexhibiting various grades or scores are shown. In an embodiment, imagedata is displayed via a graphical user interface such as that shown inFIG. 6. In an embodiment, as further discussed below, image data isdisplayed for evaluation, re-evaluation and operator input regardingtumor classification and/or image features.

Lesion Classification

In an embodiment, once image segmentation (205) is complete,classification may occur either by computer-generated classification(211), operator classification (221), or both. In an embodiment, imagedata need not be displayed to an operator (207) before computerclassification/grading may occur (211).

In an embodiment, internal, periphery and external findings may be usedto classify a lesion. In an embodiment, the interior region and theperipheral region of a lesion may be used to classify the lesion. In anembodiment, a plurality of features may graded using a scale, such as anordinal scale. In an embodiment, a vector formed from the separatelygraded features corresponds to a likely classification or diagnosis. Inan embodiment, multiple possible feature vectors can suggest a singleclassification.

In an embodiment, classification is done by assessing six specificfeatures of optoacoustic images or other parametric maps on an ordinalscale, namely:

-   -   1) internal vascularity and de-oxygenation,    -   2) peritumoral boundary zone vascularity and deoxygenation,    -   3) internal deoxygenated blush,    -   4) internal total blood,    -   5) external peritumoral radiating vessels, and    -   6) interfering artifact.        In an embodiment, the six specific features are graded on an        ordinal scale from 0-5. In an embodiment, the one or more        features are graded on an ordinal scale from 0-6. Particular        vectors of these feature scores have been shown to correlate        with particular lesion classifications. In an embodiment,        feature grades are summed to obtain a total internal score, a        total external score, and/or a total overall score. In        embodiment, a two-sided exact Jonckheere-Terpstra test is used        to test the relationship between increasing scores (internal,        external, total) and higher cancer grade.

In an embodiment, other features can be graded in addition to, or inlieu of one or more of the six specific features identified aboveincluding, but not limited to, internal, peritumoral, and peripheral:

-   -   a) vascularity;    -   b) density of vascularity;    -   c) oxygenation;    -   d) speckle;    -   e) blush;    -   f) amount of hemoglobin;    -   g) amount of blood;    -   h) ratio of oxygenated to deoxygenated blood;    -   i) blood oxygen saturation;    -   j) total blood accumulation; and    -   k) amount of interfering artifacts.

Additional features can be evaluated in both the peritumoral andperipheral region including, but are not limited to:

-   -   l) amount of tumor neovessels;    -   m) amount of vessels oriented substantially parallel to a        surface of the tumor;    -   n) amount of vessels oriented substantially perpendicular to a        surface of the tumor;    -   o) length of vessels; and    -   p) straightness of vessels.

Additional peritumoral features can be evaluated including, but notlimited to:

-   -   q) amount of proliferating tumor cells;    -   r) amount of invading tumor cells;    -   s) amount of tumor associated macrophages;    -   t) amount of native cells that have been affected by the tumor;    -   u) amount of lymphocytes;    -   v) amount of desmoplasia;    -   w) amount of edema;    -   x) amount of proteinaceous debris;    -   y) thickness of boundary zone; and    -   z) amount of tumor associated collage type 3 fibers oriented        substantially perpendicular to a surface of the tumor.

Additional peripheral features can be evaluated including, but notlimited to:

-   -   aa) amount of radiating arteries; and    -   bb) amount of radiating veins.        In an embodiment, molecular indicators are graded in addition to        grading some or all of the features identified above.

Operator Feature Grading

In an embodiment, at 221, an operator, generally a radiologist ispresented with image data related to a lesion, and is prompted to entera score for one of more features related to the lesion. In anembodiment, the image data presented to the user comprises one image. Inan embodiment, the image data may comprise a plurality of images. In anembodiment where the image data is made up of a plurality of images, itis convenient to have the plurality of images co-registered. In anembodiment, the image data comprises radiological information of avolume of tissue. In an embodiment, the images of the image data depictvisible functional or morphological structures in the volume (as theyare available to be depicted by the modality of each image). In anembodiment, the image data comprises six images as generally reflectedin FIG. 4. In an embodiment, the image data comprises boundary curvesand/or other annotations superimposed one or more images of the volumeas depicted in FIGS. 3 and 5.

In an embodiment, the operator is presented with a graphical userinterface (“GUI”) such as the interface reflected in FIG. 6. In anembodiment, the GUI includes an interface for feature grading. In theembodiment shown in FIG. 6, the interface for feature grading appearsalong the bottom portion of the GUI and allows the user to input gradesfor each of six features related to the lesion (grades for additional orfewer features can be solicited). In an embodiment, the operator mayprovide input on the grading of one or more features of the lesion viathe interface. In an embodiment, the operator selects or inputs anordinal grade for one or more features on a scale of 0-5 (other scalesmay be used). In an embodiment, the system presents the operator withone or more suggested feature grades based on analysis previouslyperformed by the system and/or another user. In an embodiment, theoperator can confirm or modify the suggested feature grades. In anembodiment, to change the focus of user input, the operator may click onan area of the screen devoted to receiving input for a particularfeature or the operator may tab between feature inputs.

In an embodiment, in addition to the image data, the operator is alsopresented with example images depicting one or more grades for aparticular feature. In the embodiment shown in FIG. 6, the exampleimages appear along the right portion of the GUI and depict ordinalgrades from 0 to 5. In an embodiment, the example images show lesionsexhibiting possible grades for the feature. The examples may includeimage data previously collected from the current patient or othersubjects. In an embodiment, the examples include images a subject matterexpert or other user previously used to evaluate the same feature (inthis or other volumes of tissue). In an embodiment, the system displaysimages that produced a score matching or similar to the system'scalculated grading of the feature. In the embodiment, the systemdisplays images that produced a score matching or similar to theoperator's inputted grade of the feature. In embodiment, the examplesinclude illustrations showing idealized presentations of the featuregrades. In an embodiment, the examples depict only a portion of thelesion such as the portion relevant to the feature being graded. Forinstance, for internal deoxygenated blush the examples may just depictthe area internal to the example lesions. Or the examples may justdepict the portion of the internal area exhibiting blush. In anembodiment, the examples shown depend on the particular feature forwhich the system is currently seeking a grade (i.e., the featurecurrently in focus). In an embodiment, the operator may indicate thegrading of the feature currently in focus by clicking on or otherwiseselecting one of the depicted examples. In an embodiment, the examplesshown change as the operator tabs or otherwise selects a differentfeature for grading. In an embodiment, the same example images are usedbut annotations (such as highlighting) are added to emphasizeinformation relevant to the feature currently in focus.

In an embodiment, an operator applies guidelines for interpretation ofimage data. In an embodiment, such guidelines may be based on example orreference images as further exemplified below with reference to FIGS.7-12. The reference images and guidelines discussed here areillustrative examples. Other types of images, guidelines, and featuresmay be used.

FIG. 7 shows reference images for internal vascularity grades 0-5 oncombined optoacoustic maps. In an illustrative embodiment, grade 0 ischaracterized by no internal vessels; grade 1 is characterized by up twointernal vessels with no more than one red vessel (indicating a vesselcarrying deoxygenated hemoglobin); grade 2 is characterized by up twointernal vessels with branches and all or most branches being green(indicating a vessel carrying oxygenated hemoglobin); grade 3 ischaracterized by internal speckle with the amount of internal green andred speckle being substantially equal and less than the amount ofexterior speckle; grade 4 is characterized by moderate internal specklewith the amount of internal red speckle being greater than the amount ofinternal green speckle and the amount of internal red speckle beinggreater than the amount of exterior red speckle; grade 5 ischaracterized by multiple internal red vessels.

FIG. 8 shows reference images for internal blush grades 0-5 on relativeoptoacoustic maps. In an illustrative embodiment, grade 0 ischaracterized by no internal vessels; grade 1 is characterized byminimal internal speckle all of which is green; grade 2 is characterizedby mild internal speckle with green and red speckle being substantiallyequal and both red and green internal speckle together being less thanor equal to the amount of exterior speckle; grade 3 is characterized bymild internal speckle with the amount of internal red speckle beinggreater than the amount of internal green speckle and both red and greeninternal speckle together being less than or equal to the amount ofexterior speckle; grade 4 is characterized by moderate internal specklewith the amount of internal red speckle being greater than the amount ofinternal green speckle and the amount of internal red speckle beinggreater than the amount of exterior red speckle; grade 5 ischaracterized by internal red blush almost filling the internal zone.

FIG. 9 shows reference images for internal hemoglobin grades 0-5 ontotal hemoglobin maps. In an illustrative embodiment, grade 0 ischaracterized by no internal vessels; grade 1 is characterized byminimal internal hemoglobin which is less than or equal to externalhemoglobin; grade 2 is characterized by a minimal number of internaldiscrete vessels with internal vascularity substantially equal toexterior vascularity; grade 3 is characterized by a moderate number ofinternal discrete vessels with internal vascularity substantially equalto exterior vascularity; grade 4 is characterized by many large internalvessels with internal vascularity greater than exterior vascularity;grade 5 is characterized by many large and heterogeneous vessels almostfilling the internal zone.

FIG. 10 shows reference images for capsular/boundary zone vessel grades0-6 shown on a various optoacoustic maps. In an illustrative embodiment,grade 0 is characterized by no capsular vessels (vessels orientedparallel to the surface of the tumor); grade 1 is characterized by up totwo capsular vessels with at least one green vessel; grade 2 ischaracterized by up to two capsular vessels with normal tapering,acutely angled branches, and mostly green; grade 3 is characterized byboundary zone speckle with green and red speckle being substantiallyequal and both red and green boundary zone speckle together being lessthan or equal to the amount of exterior speckle; grade 4 ischaracterized by boundary zone speckle with the amount of red specklebeing greater than the amount of green speckle and the amount ofboundary zone red speckle being greater than the amount of exterior redspeckle; grade 5 is characterized by three or more red boundary zonevessels; grade 6 is characterized by boundary zone blush.

FIG. 11 shows reference images for peripheral vessels grades 0-5 shownon a various optoacoustic maps. In an illustrative embodiment, grade 0is characterized by no peritumoral vessels; grade 1 is characterized byup to two peritumoral vessels with at least one green vessel; grade 2 ischaracterized by more than two peritumoral vessels with randomorientation (not radiating perpendicular to surface of lesion); grade 3is characterized by one or two radiating peritumoral vessels; grade 4 ischaracterized by more than two radiating vessels on one side of thelesion; grade 5 is characterized by more than two radiating vessels onmore than one side of the lesion.

FIG. 12 shows reference images for interfering artifacts grades 0-5shown on relative optoacoustic maps. In an illustrative embodiment,grade 0 is characterized by no significant artifacts; grade 1 ischaracterized by minimal artifacts, which do not interfere with grading;grade 2 is characterized by moderate artifacts, which do not interferewith grading; grade 3 is characterized by moderate artifacts, whichinterfere with grading; grade 4 is characterized by severe artifacts,which interfere with grading; grade 5 is characterized by severeartifacts, which make images uninterpretable.

In an embodiment, the image data or example images presented to the userare modified based on the input received from the user. In anembodiment, the image data presented to the user is modified orannotated based on the feature currently in focus. For instance, aninterfering artifact identified by the system may be highlighted orradiating vessels detected on the peritumoral boundary may be annotated.

Operator Lesion Classification

In an embodiment, the operator is presented with an interface for lesionclassification. In an embodiment, at 221, an operator, generally aradiologist is presented with image data, and is prompted to enter aclassification. In an embodiment, the operator enters a classificationof the lesion by entering text or an abbreviation for the chosenclassification. In an embodiment, the operator selects a classificationfrom a drop-down menu. Other data entry methods are known in the art andmay be used. In an embodiment, the system presents one or more possibleclassifications of the lesion based on a plurality of feature gradesentered by the operator. In an embodiment, the system presents one ormore possible classifications of the lesion based on analysis previouslyperformed by the system and/or another user. In an embodiment, theoperator is able to select, confirm, or modify a lesion classificationsuggested by the system.

Automated Lesion Classification

In an embodiment, at a 211, the diagnostic vector classification andsupport system may determine a predicted value for one or more of theplurality of features that can be graded. In an embodiment, thediagnostic vector classification and support system may determine apredicted value for the six specific features identified above. Avariety of different approaches may be taken. In an embodiment, imageprocessing or other techniques are used to mimic some or all of theoperator classification and grading techniques discussed above. In anembodiment, such techniques are applied in an attempt to reach the sameor similar results by different means. In an embodiment, such techniquesare applied toward different objectives. The techniques discussed beloware illustrative examples. Other types of techniques may be used.

In an embodiment, a hemoglobin-like parametric image and anoxygenation-like parametric image are used. Such images are referred toin this section as processed images. A processed image may be filteredby one or more appropriate filters prior to feature detection. In anembodiment, one appropriate filter is a smoothing filter. In anembodiment, one appropriate filter is a shape detection filter, wherebywhen the shape to be detected is centered about a pixel, the filterresults in a high intensity for that pixel, and when this is not thecase, the intensity produced in the filtered image is low. In anembodiment, the shape detection filter is optimized to detect vessels.In an embodiment, a shape filter may be directional or include furtherdirectional information such (e.g. angle of a line or vessel). Becauseradiating vessels that radiate from the peritumoral region into theperipheral region may pass through the peritumoral boundary, and tend tobe directed more perpendicular rather than tangent to the secondboundary, a directional filter may be used to detect this condition. Inan embodiment, more than one shape filter can be used.

Vascularity may be determined from a processed hemoglobin image.Oxygenation or deoxygenation may be determined from a processedoxygenation image. Thus, features involving vascularity may be foundfrom the processed hemoglobin image. Features involving oxygenation maybe found from the processed oxygenation image. Finally, a combined imageparametrically reflective of deoxygenated hemoglobin masked using theimage parametrically reflective of total hemoglobin (e.g., image 460)may be used, instead of, or in addition to the processed oxygenationimage, to predict feature grades related to oxygenation of vessels.

To determine metrics used to quantify the presence of features for asegmented lesion, the internal, peripheral, and external region adjacentto the periphery may be used.

In an embodiment, internal deoxygenated blush is measured by determiningthe amount of pixels reflecting deoxygenation in the interior region. Inan embodiment, the deoxygenated blush grade may be determined as aresult of calculating the number of pixels that reflect deoxygenationbeyond a threshold in total, or as a weighted total. In an embodiment,the deoxygenated blush grade may be determined as a result of theproportion (e.g. percentage) of the total number of pixels of theinterior region that reflect deoxygenation, or deoxygenation beyond athreshold.

A parametric image or parametric image overlay may use color toillustrate a parameter. In an embodiment, the parametric image overlay(shown in image 460), can use red colors to indicate areas comprisingone functional determination, i.e., concentration of deoxygenatedhemoglobin, and green colors to indicate a different functionaldetermination, i.e., areas comprising a concentration of oxygenatedhemoglobin. In an embodiment, the number of red colored pixels and thenumber of green colored pixels may be used in lesion classification,such as grading internal deoxygenated blush. For example, in anembodiment, a weighted version of the number of internal red pixels andinternal green pixels (including information about how red or how greeneach pixel is) may be used to produce total internal redness (weightedsum of pixels more red than a threshold), total internal greenness(weighted sum of pixels more red than a threshold), and/or a totalinternal metric (weighted sum of all the pixels, green positive weightand red negative weight). A ratio of internal red pixels to internalgreen pixels, or total redness to total greenness may be used in gradingthe internal deoxygenated blush.

Peritumoral boundary zone vascularity and deoxygenation may be computedby performing similar functions in the peritumoral region.

In an embodiment, other molecular indicators (beyond hemoglobin andoxygenation) may be used. In an embodiment, other molecular indicatorscan be determined by using different or additional predominantwavelengths to generate the stimulated response leading to theoptoacoustic image.

The techniques described above may be applied to absolute contrast or(as discussed below, relative contrast) determined on the basis of thedetermined oxygenation and/or hemoglobin (and/or other such molecular)metric. In an embodiment, a region of interest may be used to improvecontrast. Using a region of interest (as generally described in U.S.patent application Ser. No. 13/793,808, filed Mar. 11, 2013 and entitled“Statistical Mapping in an Optoacoustic Imaging System”) positionedproximate to or over the lesion, may cause the colorization of theinternal, periphery and external parametric image data to become moreappropriate for application of the above techniques relying oncolorization. Thus, the characterization techniques above may be appliedon relative contrast based on statistical properties of the tissue. Whenrelative contrast is used, the weights as described above can bedetermined in relation to the reference level. In an embodiment, thereference level may correspond to a weight of zero. In an embodiment, aweight of one (+1) and negative-one (−1) may correspond to values aboveand below the reference level. In an embodiment, the image amplitudecorresponding to unity weighting magnitude (+1 or −1) may be fixed, ormay be based on the statistical properties of the tissue (e.g. inproportion the standard deviation of the region of interest). In anexemplary embodiment, +1 corresponds to a pixel having its imageintensity less the reference level equal to K standard deviations. In anembodiment, the reference level may be the mean of the region ofinterest.

Pattern classification filters may be used. In an embodiment, an imageis converted to a pattern classification domain, such as a 2D waveletpacket domain. Apriori knowledge indicating when the spatialcoefficients of the pattern classification domain indicate the presenceof a feature in such a filter may be learned by a training phase of analgorithm. In an embodiment, such a technique uses a support vectormachine (SVM), or other similar method for finding pattern clusters toproduce a classifier, or other such technique. Thus, the presence ofsuch features in an image may be quantified spatially on a per-pixelbasis, and methods for counting the occurrence of such quantifiedmeasures within the defined boundaries of an image segment may be usedto assess features in that zone of the image.

Artifacts, such as streaking, interferes with the determination ofvessels as such artifacts may, e.g., mimic vessels. In an embodiment, afilter may be employed to suppress streaking artifact, or filter outsuch artifact. In an embodiment, the amount of such artifact as detectedmay be quantified by the filter and used as a criterion in the techniquedescribed above. In an embodiment, iterative reconstruction processingmay be used to remove streaking artifacts. Many other techniques forremoving artifacts from images are known in the art and can be appliedby one skilled in the art.

Accordingly, in an embodiment, to compute the six features, one or moreof the above-described techniques can be used:

-   -   1) internal vascularity and de-oxygenation: a score is based on        the vessels detected in the hemoglobin image within the first        boundary, and how oxygenated these vessels are from the        oxygenation image. In an embodiment, a combined image (e.g.,        FIG. 4, 460) may be used. In an embodiment, a vessel detector        may be used. In an embodiment, vascularity may be inferred from        the amount of hemoglobin. In an embodiment, the score is related        to the ratio of redness to greenness in the combined image.    -   2) peritumoral boundary zone vascularity and deoxygenation: a        score is based on the vessels detected in the hemoglobin image        within the peritumoral boundary (i.e., between the first and        second boundary), and how oxygenated these vessels are from the        oxygenation image. In an embodiment, a combined image (e.g.,        FIG. 4, 460) may be used. In an embodiment, a vessel detector        may be used. In an embodiment, vascularity may be inferred from        the amount of hemoglobin. In an embodiment, the score is related        to the ratio of redness to greenness in the combined image.    -   3) internal deoxygenated blush: a score is determined as        described above, for the internal region from the oxygenation        image. In an embodiment, the score is related to the percentage        of red pixels of the processed oxygenation map (e.g., FIG. 4,        450).    -   4) internal total blood: a score is determined as described        above, in the internal region, from the hemoglobin image        intensity, or vascular detected image. In an embodiment, the        score is related to the percentage of internal pixels exceeding        a threshold using the processed hemoglobin map (e.g., FIG. 4,        440).    -   5) external peritumoral radiating vessels: a score is determined        from radiating vessels detected on the peritumoral boundary. In        an embodiment, the score is related to the sum of directional        filtered hemoglobin image proximate to external boundary, where        such vessels near perpendicular directions to the boundary are        scored high and other features are suppressed.    -   6) interfering artifact: a score is determined as described        above. In an embodiment, artifacts are removed prior to scoring        and thus, the interfering artifact score is zero.

In an embodiment, each of the foregoing features is scored on a 0-5ordinal scale. In an embodiment, the presence of capsular/boundary zonevessels is scored on a 0-6 ordinal scale. In an embodiment, features forthe ordinal scale may involve complex logic which includes conditionalstatements (e.g. “if”) to describe ranking on the ordinal scale incertain circumstances and can use more than one such metric asdescribed.

In an embodiment, a classification vector is formed by the scoring ofeach feature. The classification vector corresponds to a prediction ofthe classification for the lesion. In an embodiment, the classificationvectors are determined empirically, by comparing computer scores for thefeatures with histological data for a population of samples. Using thisempirical method, classification vectors, which represent a summary fora population, can be updated as new classifications and histologicalinformation are available.

Diagnostic Support

In an embodiment, at 231, a classification received from a user iscompared to a classification calculated by the system. In an embodiment,if the classifications differ or differ by a threshold degree,diagnostic support is provided to the user at 235. In an embodiment, ifthe operator and computer-generated classifications are the same ordiffer only by a threshold degree, the operator feature classificationis output at 241. In an embodiment, diagnostic support is offered evenwhere the operator and computer-generated classifications are alreadythe same.

In an embodiment, at 231, a feature grade received from a user can alsobe compared to a feature grade calculated by the system. If the featuregrades differ or differ by a threshold degree, diagnostic support may beprovided to the user at 235. In an embodiment, diagnostic support isoffered even where the operator and computer-generated grades arealready the same. In an embodiment, if the operator andcomputer-generated grades are the same or differ only by a thresholddegree, the operator feature grade is output at 241. In an embodiment,where the difference in operator and computer-generated feature gradeswould not affect the resulting classification of the tumor, diagnosticsupport is not offered and the method proceeds to 241.

As further discussed below, diagnostic support 235 may includepresenting additional information to the user, soliciting additionalinput from the user, or both. In an embodiment, less information ispresented to the user to focus the user on particular information.

In an embodiment, where a classification or feature grade received fromthe user differs or differs substantially from a classification orfeature grade calculated by the system, the system presents additionalimage data, example images, or other information to the operator. Forexample, in an embodiment, the system highlights or otherwise annotatesthe image data displayed to emphasize information that formed the basisof the system's classification or grading. In an embodiment, the systemdisplays a subset or portion of the image data to focus the operator oninformation that formed the basis of the system's classification orgrading. In an embodiment, the system displays additional images to theoperator. For example, in an embodiment, the system displays exampleimages as discussed above. In an embodiment, the system displaysadditional image data to the operator.

In an embodiment, where a classification or feature grade received fromthe user differs or differs substantially from a classification orfeature grade calculated by the system, the system solicits additionalinput from the operator. Such solicitation may occur before, after,during, or instead of the presentation of additional information to theoperator. For example, in an embodiment, the operator is asked to gradeor re-grade one or more features of the lesion. In an embodiment, theoperator is asked to select portions of the image that formed the basisof the operator's grading of the feature. In an embodiment, the operatoris asked to confirm, modify, or augment first or second boundary curvesand/or segmentation of images. In an embodiment, the operator mayprovide such additional information regardless of whether the operator'sclassification or feature grades differ from those calculated by thesystem. In an embodiment, the system solicits such additionalinformation regardless of whether the operator's classification orfeature grades differ from those calculated by the system. In anembodiment, the operator may provide such additional informationregardless of whether the system solicits it.

In an embodiment, the system may then re-evaluate one or more featuresof the lesion and/or its classification of the lesion based on anyadditional information provided by the operator, for example a modifiedboundary, image segmentation, or feature grade. The system may then onceagain display additional image data or solicit additional input from theoperator.

In an embodiment, the operator may confirm the operator's classificationor feature grades during diagnostic support 235. In an embodiment,reconfirmation of causes the method to terminate by returning theconfirmed conclusions at 241. In an embodiment, the system requires twoor more confirmations or re-confirmations before the method terminateswith the confirmed values.

In an embodiment, the operator modifies the operator's classification orfeature grades during diagnostic support 235 and the method returns to231 to compare the modified classification or feature grade to theclassification or feature grades computed by the system. If the modifiedoperator classification or feature grade now match or substantiallymatch the computed classification or feature grade, the methodterminates by returning the modified operator classification or featuregrade at 241.

In an embodiment, the operator confirms one or more of the computedclassifications or feature grades, which causes a positive comparison at231, and termination with return of the confirmed classification orfeature grade at 241.

Study

A further illustrative embodiment is described below with reference to astudy using opto-acoustics (OA), a dual energy laser technologyco-registered with diagnostic ultrasound to simultaneously assessstructural and functional features of breast masses. OA requires noinjected agent and utilizes no radiation. A description of anoptoacoustic device and its features can be found in the Parametric MapApplication.

A study was conducted concerning a new method and system for how to useOA features to classify breast masses as malignant or benign. In anembodiment, six specific OA features were assessed using a 0-5 ordinalscale:

-   -   1) internal vascularity and de-oxygenation,    -   2) peri-tumoral boundary zone vascularity and deoxygenation,    -   3) internal deoxygenated blush,    -   4) internal total blood,    -   5) external peri-tumoral radiating vessels, and    -   6) interfering artifact.

Analyses were performed using: Logistic Regression (LR), Support VectorMachines (SVM), Classification Trees (CT), Random Forests (RF), andK-Nearest Neighbors (KNN). Ten-fold cross validation was used, where the66 cases were randomly divided into 10 groups. Each group was removed inturn from the 66 observations and the classifier was trained on theremaining groups to develop a classification rule. This rule was thenapplied to the removed group. This was repeated 10 times until everyobservation was assigned by a classifier that had not previously beendeveloped using that observation.

Results

Sensitivity and specificity assessments by each method are summarized inthe table below. KNN and SVM performed the best, while LR performed theworst, but the results were consistent and favorable for all 5 analyses;sensitivity ranged from 95% to 100% and specificity ranged from 52% to79%. Findings support the biopsy-sparing potential for OA imaging.

Malignant Benign Classified Classified as as Method Benign CancerSensitivity Specificity Logistic Regression 2 14 95% 52% Support VectorMachine 0 6 100% 79% Classification Trees 1 10 97% 66% Random Forest 0 8100% 72% K Nearest Neighbor 0 6 100% 79%

Data suggest that new method of analysis of breast masses using OA imagecan achieve clinically meaningful sensitivity and specificity in adiagnostic setting.

Summary

Analysis yields sets of features that can cluster the data (only IDC andFA were examined) in to three clusters: cluster #1 is FA, cluster #2 isIDC-GR1, cluster #3 is IDC-GR2, IDC-GR3 (and a small number of ILC-GR2).In general, this is consistent with the rules as proposed by Dr.Stavros. This report may further formalize that analysis, or providemore insight into the use of the rule sets.

Method

Features were selected and ranked (from values 1 to 5). The rankingcriterion used was shown in FIGS. 13A and 13B. An analysis was done onFA and IDC-GR1, IDC-GR2 and IDC-GR3 patients to determine what criteriaare significant to distinguish the two classes. The matlab anovaanalysis tool was used. The results is presented in an informal format.

Data

The data contained graded contained a subset 80 patients. Of these 80patients, there were 13 that were not graded yet (incomplete). Of the80, there were 28 FA, 3 ILC and 48 IDC. There were 51 malignants and 29benigns. For the malignants, there were 11 GR1, 18 GR2, and 22GR3. Datafrom the other types of lesions encountered in the feasibility studywere not yet graded on the spreadsheet for input to this analysis, andhence was not analyzed.

Observations

FIGS. 14A and 14B show a scatter plot of the features. Each dotrepresents a patient. The red dots are benign patients. The benignpatients cluster into the bottom left of the image. The x-axisrepresents a score based on feature vector c2. The y-axis represents ascore based on feature vector c1. Each feature vector represents acombination of features where the weights were solved by the ANOVAanalysis. Feature vectors c1 and c2 are shown in FIG. 15. In FIG. 2b ,the patient IDs of the outliers is shown.

From FIG. 15, feature vector c2 detects if patient contains mainly thefollowing: low internal vv, low boundary vv, low internal blush, lowinternal blood, and permitting a small amount of surround vv. If thecombination of these features are mainly present, then the c2 score willindicate that the diagnosis is likely not an IDC-GR3 when choosingbetween a FA and an IDC. Feature vector c2 was the second best predictordetermined from the ANOVA analysis. However, feature vector c2 in isalso able to cluster the IDC-GR1s apart from the benigns on the x-axis.

Also from FIG. 15, feature vector c1 (the highest predictor) detects ifa patient contains mainly the following: any internal blush (internalblush is most highly weighted in the graph), significant surround vv,and a large amount of internal blood. With a low score on feature vectorc1, IDC-GR1 and IDC-GR2 and ILC-GR2 can be separated from the categoryof both FA and IDC-GR1 (y-axis).

Conclusions

The analysis yields sets of features that can cluster the data (whenchoosing between IDC and FA) in to three clusters: cluster #1 is FA,cluster #2 is IDC-GR1, cluster #3 is IDC-GR2, IDC-GR3 and ILC-GR2. Ingeneral, this is consistent with the rules as proposed by Dr. Stavros.Complimentary information may be yielded in this analysis.

The features listed in c1 and c2 may be used to assist in the diagnosisof OA information.

Patients primarily where patient contains mainly the following (c2vector): low internal vv, low boundary vv, low internal blush, lowinternal blood, and permitting a small amount of surround vv can begrouped into a class that will distinguish IDC-GR1 from other classes.Patients images that contain mainly the following: any internal blush(internal blush is most highly weighted in the graph), significantsurround vv, and a large amount of internal blood can be grouped into aclass that will distinguish IDC-GR1 from other classes.

Note

Ultrasound features were not considered in this analysis. In many casesultrasound features may be necessary to distinguish different lesions,and determine under what situations optoacoustic features areapplicable.

Those skilled in the art will recognize that the methods and systems ofthe present disclosure may be implemented in many manners and as suchare not to be limited by the foregoing exemplary embodiments andexamples. In other words, functional elements being performed by singleor multiple components (or sub-systems), in various combinations ofhardware and software or firmware, and individual functions, may bedistributed among software applications at either the client level orserver level or both. In this regard, any number of the features of thedifferent embodiments described herein may be combined into single ormultiple embodiments, and alternate embodiments having fewer than, ormore than, all of the features described herein are possible.Functionality may also be, in whole or in part, distributed amongmultiple components, in manners now known or to become known. Thus,myriad software/hardware/firmware combinations are possible in achievingthe functions, sub-systems, features, interfaces and preferencesdescribed herein. Moreover, the scope of the present disclosure coversconventionally known manners for carrying out the described features andfunctions and interfaces, as well as those variations and modificationsthat may be made to the hardware or software or firmware componentsdescribed herein as would be understood by those skilled in the art nowand hereafter.

Various modifications and alterations to the invention will becomeapparent to those skilled in the art without departing from the scopeand spirit of this invention. It should be understood that the inventionis not intended to be unduly limited by the specific embodiments andexamples set forth herein, and that such embodiments and examples arepresented merely to illustrate the invention, with the scope of theinvention intended to be limited only by the claims attached hereto.Thus, while the invention has been particularly shown and described withreference to a preferred embodiment thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the spirit and scope of theinvention.

The invention claimed is:
 1. A method for providing support inclassifying a lesion using an optoacoustic image of a volume of tissue,wherein the volume of tissue comprises a tumor having a central nidus,the method comprising: obtaining an ultrasound image of the volume oftissue, the ultrasound image presenting at least part of the centralnidus and at least part of a peritumoral region; obtaining anoptoacoustic image of the volume of tissue, the optoacoustic image beingcoregistered with the ultrasound image and presenting the opticalcontrast of at least a portion of the part of the central nidus and atleast a portion of the part of the peritumoral region; identifying onthe ultrasound image a tumoral boundary curve, the tumoral boundarycurve approximating at least a portion of a perimeter of the centralnidus of the tumor; presenting on a display at least a portion of theoptoacoustic image with the tumoral boundary curve superimposed thereon;defining an internal zone within the displayed image based on thetumoral boundary curve; and obtaining from an operator an operatorfeature score for a tumoral feature contained within the internal zone;obtaining from the operator an operator feature score for anextra-tumoral feature contained at least partially outside the internalzone; calculating by computer one or more computer-generated featurescores for the tumoral feature, based at least in part on informationfalling within the internal zone of the displayed image; calculating bycomputer one or more computer-generated feature scores for theextra-tumoral feature, based at least in part on information fallingoutside the internal zone of the displayed image; obtaining one or moresupplementary inputs from the operator if either of the operator featurescores differ from the corresponding at least one computer-generatedfeature scores; and classifying the lesion based on the operator featurescores.
 2. The method of claim 1, wherein at least one of the one ormore supplementary inputs is a modification to the operator featurescore, the method further comprising re-obtaining the one or moresupplementary inputs from the operator if the operator feature scorediffers from at least one of the one or more computer-generated featurescores.
 3. The method of claim 1, wherein the at least one of the one ormore supplementary inputs is a confirmation of the operator featurescore.
 4. A method for providing support in classifying a lesion usingan optoacoustic image of a volume of tissue, wherein the volume oftissue comprises a tumor having a central nidus, the method comprising:obtaining an ultrasound image of the volume of tissue, the ultrasoundimage presenting at least part of the central nidus and at least part ofa peritumoral region; obtaining an optoacoustic image of the volume oftissue, the optoacoustic image being coregistered with the ultrasoundimage and presenting the optical contrast of at least a portion of thepart of the central nidus and at least a portion of the part of theperitumoral region; identifying on the ultrasound image a tumoralboundary curve, the tumoral boundary curve approximating at least aportion of a perimeter of the central nidus of the tumor; presenting ona display at least a portion of the optoacoustic image with the tumoralboundary curve superimposed thereon; defining an internal zone withinthe displayed image based on the tumoral boundary curve; and obtainingfrom an operator a classification for the lesion; calculating bycomputer one or more computer-generated feature scores for the tumoralfeature, based at least in part on information falling within theinternal zone of the displayed image; calculating by computer one ormore computer-generated feature scores for the extra-tumoral feature,based at least in part on information falling outside the internal zoneof the displayed image; determining one or more computer-generatedclassifications for the lesion based upon at least the one or morecomputer-generated feature scores for the extra-tumoral feature and theat least one or more computer-generated feature scores for the tumoralfeature; obtaining one or more supplementary inputs from the operator ifthe operator classification differs from all of the one or morecomputer-generated classifications for the lesion; and classifying thelesion based on the operator classification for the lesion.
 5. A methodfor providing support for classifying a tumor having a central nidus anda peritumoral region adjacent to the central nidus, the methodcomprising: presenting on a display an optoacoustic image of at least aportion of a volume of tissue comprising the tumor; identifying on thedisplay an internal zone of the optoacoustic image approximating atleast a portion of the central nidus of the tumor; evaluating at least aportion of the optoacoustic image falling within the internal zone todetermine a grade for at least one internal feature contained within thecentral nidus of the tumor; identifying on the display a boundary zoneof the optoacoustic image approximating at least a portion of theperitumoral region of the tumor; evaluating at least a portion of theoptoacoustic image falling within the boundary zone to determine a gradefor at least one peritumoral feature contained at least partially withinthe peritumoral region of the tumor; and evaluating a portion of theoptoacoustic image falling outside the internal zone and outside theboundary zone to determine a grade for at least one peripheral featureexternal to the central nidus of the tumor and at least partiallyexternal to the peritumoral region of the tumor; identifying aclassification of the tumor based on the at least one internal featuregrade, the at least one peritumoral feature grade, and the at least oneperipheral feature grade, wherein determining a grade comprises:calculating by computer one or more computer-generated feature scoresfor the at least one feature; obtaining an operator feature score forthe at least one feature; and comparing at least one of the one or morecomputer-generated feature scores to the operator feature score; whereinthe at least one peripheral feature is selected from the set of: a)vascularity; b) oxygenation; c) speckle; d) blush; e) amount ofhemoglobin; f) amount of blood; g) ratio of oxygenated to deoxygenatedblood; and h) amount of radiating arteries; i) amount of radiatingveins; j) amount of tumor neovessels; k) amount of vessels orientedsubstantially parallel to a surface of the tumor; l) amount of vesselsoriented substantially perpendicular to a surface of the tumor; m)length of vessels; n) straightness of vessels; and o) amount ofinterfering artifacts.